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PREFACE 


This  study  was  performed  under  work  unit  71840919  in  support  of  the  Supervisory  Control 
Infonnation  Management  Research  (SCIMR)  program.  The  objective  of  this  study  was  to 
investigate  the  impact  and  effectiveness  of  various  commercial,  off-the-shelf  input  devices  as 
alternatives  to  the  standard  mouse  for  simple  computer  screen  movements  and  interactions.  This 
study  will  serve  as  the  foundation  for  exploration  into  multi-modal  input  device  efficacy  for 
multi-remotely  piloted  aircraft  (RPA)  supervisory  control. 


1.0  INTRODUCTION 


Military  operations  revolve  around  intelligence  gathering,  integration,  and  distribution  to 
enable  warfighters  to  successfully  accomplish  their  missions.  Key  components  in  the 
reconnaissance,  surveillance,  and  target  acquisition  effort  are  remotely  piloted  aircraft. 

Currently,  RPAs  require  multiple  personnel  to  directly  pilot  and  monitor  the  sensor  feeds  of  one 
aircraft.  The  Supervisory  Control  Interfaces  Branch  at  the  Air  Force  Research  Laboratory  is 
working  to  enable  one  operator  to  successfully  manage  and  operate  the  payloads  of  multiple 
autonomous  RPAs  simultaneously.  The  SCIMR  program  is  investigating  infonnation 
management  aspects  for  a  single  operator  supervising  multiple  RPAs.  The  program’s  objective 
is  to  investigate,  develop,  and  evaluate  novel  information  management  tools  (controls,  displays, 
and  decision  aids)  to  support  customer  requirements  while  maintaining  appropriate  RPA  operator 
situational  awareness,  workload,  and  performance.  The  current  study  is  concerned  with 
examination  of  the  utility  of  a  variety  of  input  controllers  for  perfonning  four  common  actions 
associated  with  foreseeable  multiple-RPA  control  when  using  a  desktop  computer  as  a  ground 
control  station. 

Methods  for  analysis  of  input  methods  fall  into  four  categories:  completion  time,  degree 
of  error,  training  time,  and  ergonomics  (Zhai,  2009).  Fitts’  law  (Fitts,  1954)  is  a  commonly-used 
model  of  human  movement  in  ergonomics  and  human-computer  interaction  used  to  study  input 
device  perfonnance.  It  predicts  that  the  time  required  to  quickly  move  to  a  target  area  is  a 
function  of  the  size  and  distance  of  the  target.  Fitts’  law  is  used  to  model  the  act  of  pointing, 
either  by  physically  touching  an  object  with  a  linger  or  hand  or  virtually  by  pointing  to  an  object 
on  a  display  using  a  pointing  device  (e.g.,  mouse,  light  pen,  control  stick).  However,  it  is  of 
limited  use  for  some  complex  tasks  as  it  can  be  applied  only  to  pointing  tasks  (Accot  &  Zhai, 
1999).  The  current  study  augmented  Fitts’  law  with  the  use  of  trajectory-based  evaluations  and 
those  involving  steering  law.  Steering  tasks  and  trajectory-based  tasks  aim  to  analyze  controller 
movement  of  a  cursor  around  a  display  (Accot  &  Zhai,  1999).  These  analyses,  however,  will  be 
altered  to  be  more  aligned  with  movements  that  might  be  encountered  using  a  control  station  for 
multiple  RPAs. 


2.0  METHODS 
2.1  Participants 

The  participants  for  this  study  consisted  of  12  civilian  and  military  personnel  stationed  at 
Wright-Patterson  Air  Force  Base,  OH.  The  sample  consisted  of  6  men  and  6  women.  Ten  of  the 
12  participants  were  between  18-25  years  of  age  and  2  were  between  26-35  years.  Participants 
were  required  to  have  normal  visual  acuity  (20/20)  or  corrected-to-nonnal  visual  acuity  in  both 
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eyes  and  normal  color  vision.  Visual  acuity  and  color  vision  were  determined  by  self-report. 
Participation  was  voluntary;  no  compensation  was  offered  for  participation  in  this  study. 


2.2  Hardware 

The  test  configuration  consisted  of  a  widescreen  Dell  LCD  monitor  with  a  resolution  of 
1920  x  1200  pixels.  Six  commercial  devices  were  evaluated:  1)  standard  mouse,  2)  Belkin 
n52te,  3)  Saitek  Cyborg  Command  Unit,  4)  Wacom  Bamboo  Fun  with  stylus,  5)  Wacom 
Bamboo  Fun  with  touch,  and  6)  Xbox  360  controller.  See  Figure  1. 


Figure  1.  Control  devices  used  in  the  study.  Upper  row  (left  to  right):  standard  mouse,  Belkin  n52te, 
and  Saitek  Cyborg  Command  Unit;  lower  row  (left  to  right):  Wacom  Bamboo  Fun  (with  stylus  or  touch) 

and  Xbox  360  controller. 


2.3  Mapping 

There  were  six  functions  mapped  to  each  input  device:  cursor  movement,  “clicking”, 
“dragging”,  zoom  in,  zoom  out,  and  fast  zoom.  See  Table  1  for  a  comparison  of  function 
mapping  across  all  devices,  and  Figures  2  through  8  for  graphic  illustrations  of  the  function 
mapping. 
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Table  1.  Device  Function  Mapping. 


Device 

Cursor 

Movement 

Clicking 

Dragging 

Zoom  in 

Zoom  out 

Fast  Zoom 

Mouse 

Move 

mouse 

Left  button 

Hold  left 

button 

Scroll  forward 

on  scroll  wheel 

Scroll  backward 

on  scroll  wheel 

Depress  scroll 
wheel  while 
scrolling 

Belkin 

Press  d-pad 

10  button 

Hold  10  button 

Scroll  forward 

on  scroll  wheel 

Scroll  backward 

on  scroll  wheel 

Depress  scroll 
wheel  while 
scrolling 

Saitek 

Analog  stick 

8  button 

Hold  8  button 

Hold  15  button 

Hold  16  button 

Hold  5  button 
while  zooming 

Touch 

Slide  finger 

across 

touchpad 

Tap 

touchpad 
with  finger 

Double-tap 
touchpad  with 
finger 

Slide  two 
fingers  toward 
each  other 
across  touchpad 
(“squeezing” 
gesture) 

Slide  two  fingers 
away  from  each 
other  across 
touchpad 
(“stretching” 
gesture) 

N/A 

Stylus 

Move  stylus 
over  tablet 

Touch 
stylus  to 
tablet 

Touch  stylus 
to  tablet  and 

hold 

Press  button  on 
stylus,  slide 
stylus  down 

Press  button  on 
stylus,  slide 
stylus  up 

N/A 

Xbox  360 

Left  analog 
stick 

A-button 

Hold  A-button 

Left  bumper 
(LB) 

Right  bumper 
(RB) 

Hold  B  -button 
while  zooming 
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Figure  2.  Mouse  function  mapping. 


Figure  3.  Belkin  n52te  function  mapping. 


8  key 
(left  click) 


5  key 

(fast  zoom) 


Analog  thumb 
stick 


15  key  (lower,  zoom 
out)/16  key  (upper, 
zoom  in) 


Figure  4.  Saitek  Cyborg  Command  Unit  function  mapping. 
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Figure  5.  Wacom  Bamboo  Fun  with  stylus  and  touch  function  mapping.  Operating  the  Wacom 
Bamboo  Fun  with  the  stylus  involves  touching  the  stylus  to  the  interaction  area.  If  touch  is  being  used, 
the  operator  uses  his  or  her  fingers  to  perform  gesture  commands  (as  outlined  in  Table  1)  on  the 

interaction  area. 


Figure  6.  Xbox  360  controller  function  mapping. 
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2.4  Measures 

Data  collection  included  demographic  data,  objective  task  performance,  and  subjective 
ratings  of  the  utility  of  the  devices  for  performing  the  experimental  tasks.  Copies  of  the  study 
questionnaires  are  provided  in  Appendix  A. 

Demographic  data.  This  questionnaire  elicited  information  about  participants’  age, 
vision  (acuity,  color  vision),  video  game  experience,  and  control  device  experience.  Device 
experience  level  ratings  used  a  5-point  scale:  1-very  low,  2  -  low,  3  -  average,  4  -  high,  5  -  very 
high. 


Task  performance.  Objective  performance  for  the  neutral  point  movement,  dragging,  and 
zooming  tasks  was  average  time  to  complete  the  task  (measured  in  seconds).  Objective 
perfonnance  for  the  tracking  task  was  root  mean  square  (RMS)  error  averaged  over  several 
replications. 

Subjective  ratings.  Following  completion  of  trials  for  each  device,  participants  rated  that 
device  for  its  utility  in  perfonning  each  of  4  tasks:  neutral  point  movement,  tracking,  dragging, 
and  zooming.  Ratings  were  on  a  5-point  scale  that  ranged  from  1  “very  poor”  to  5  “very  good.” 
Participants  also  were  asked  to  provide  comments  regarding  the  strengths  and  weaknesses  of  the 
device  and  suggestions  for  modifications.  On  completion  of  the  experiment,  participants  rank- 
ordered  the  6  devices  from  1  “best”  to  6  “worst”  in  regard  to  their  perfonnance  on  each  of  the 
experimental  tasks:  neutral  point  movement,  tracking,  dragging,  and  zooming.  They  also  were 
asked  to  indicate  their  favorite/least  favorite  aspects  of  the  devices  and  what  they  would  change, 
combine,  or  remove  from  the  devices  to  improve  perfonnance  on  the  experimental  tasks. 


2.5  Procedures 

The  study  began  with  a  briefing  regarding  the  study  objectives  and  completion  of  the 
informed  consent  form  and  biographical  data  collection.  During  the  course  of  the  experiment, 
each  participant  performed  4  tasks  (neutral  point  movement,  tracking,  dragging,  and  zooming) 
with  each  on  the  6  devices.  The  order  of  the  devices  varied  across  participants.  However,  the 
order  of  the  tasks  was  the  same  for  each  device.  Participants  were  allowed  to  train  with  each 
device/task  combination  until  they  indicated  they  were  comfortable  with  the  task  and  input 
device.  Following  practice,  participants  completed  several  test  trials  for  each  task/device 
combination,  then  completed  a  post-device  questionnaire  regarding  the  utility  of  the  device  for 
perfonning  each  task  and  their  comments  regarding  the  strengths  and  weaknesses  of  that  device. 
This  procedure  was  followed  until  each  of  the  devices  had  been  evaluated.  On  completion  of  the 
test  trials,  participants  completed  the  post-study  questionnaire. 

The  tasks  were  neutral  point  movement,  tracking,  dragging,  and  zooming.  In  the  neutral 
point  movement  task  (see  Figure  7),  the  cursor  was  moved  from  the  center  of  a  black  background 
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to  a  yellow  circle  that  appeared  at  a  varying  distance  from  center  (2”  or  6”)  and  varying  angle 
(0°,  60°,  120°,  180°,  240°,  300°).  The  measure  of  interest  was  time  between  clicking  at  the 
center  of  the  screen  to  clicking  on  the  target  circle.  Reaction  time  (time  of  appearance  of  the 
yellow  circle  to  time  of  click  on  the  center  circle)  was  not  included  in  the  time  measurement. 
There  were  36  test  trials. 


TASK 

OEVICE  Ti»nng 


Figure  7.  Neutral  point  movement  task. 

In  the  tracking  task  (see  Figure  8),  a  yellow  circle  appeared  on  a  black  background  and 
moved  at  varying  speeds  in  a  predetermined  path.  The  path  consisted  of  a  string  of  shapes 
(square,  spiral  in,  spiral  out,  circle,  zigzag),  but  the  sequence  was  not  known  by  the  participants. 
The  task  was  to  maintain  the  cursor  over  the  yellow  circle  throughout  the  movement  along  the 
path.  The  measure  of  interest  was  RMS  error,  calculating  distance  of  the  cursor  from  the  edge  of 
the  circle.  There  were  5  test  trials. 
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TASK:  Track 
DEVICE:  Traning 


Figure  8.  Tracking  task. 

In  the  dragging  task  (see  Figure  9),  a  white  ring  appeared  at  the  center  of  a  black 
background.  Yellow  circles  with  a  radius  of  0.3”  appeared  6”  from  the  center  at  angles  of  0°, 
60°,  120°,  180°,  240°,  and  300°.  Participants  were  required  to  click  and  drag  the  yellow  circles 
to  the  white  ring  at  the  center  of  the  screen  and  release  the  click.  The  measure  of  interest  was  of 
time  between  click  depression  and  release  of  the  input  mechanism  (i.e.,  hotkey,  mouse  click, 
etc.).  There  were  18  test  trials. 


TASK:  Drag 
DEVICE:  Traning 


Figure  9.  Dragging  task. 
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In  the  zooming  task  (see  Figure  10),  two  circles  (one  translucent  grey,  one  yellow) 
appeared  centered  on  a  black  screen.  Participants  used  the  zooming  mechanism  to  adjust  the  size 
of  the  yellow  circle  to  match  that  of  the  target  grey  circle.  The  measure  of  interest  was  the  time 
elapsed  from  the  beginning  of  zoom  until  the  participants  matched  the  circle  sizes  and  clicked  to 
indicate  completion.  There  were  18  test  trials. 


2.6  Analyses 

In  all  analyses,  the  standard  mouse  was  treated  as  a  baseline  condition  against  which  the 
effectiveness  of  the  other  control  devices  was  evaluated.  Related  samples  t-tests  were  used  to 
examine  all  of  the  objective  measures  of  performance  and  subjective  measures  that  used  interval- 
level  scales.  The  Wilcoxin  signed  ranks  test  (Wilcoxin,  1945)  was  used  to  examine  participants’ 
subjective  rankings  of  the  devices  collected  in  the  post-study  questionnaire.  The  Wilcoxin 
signed  ranks  test  is  the  nonparametric  analogue  of  the  ordinary  2-related  samples  t-test.  It  is 
used  when  the  distributional  assumptions  that  underlie  the  t-test  cannot  be  met.  We  had  no  clear 
expectation  of  how  perfonnance  would  be  affected  by  control  device.  As  a  result,  non- 
directional  (2-tailed)  tests  were  used.  All  statistical  tests  used  a  0.05  Type  I  error  rate. 
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3.0  RESULTS 

Participants’  self-reported  level  of  experience  varied  among  the  devices.  Paired  samples 
t-tests  indicated  that  they  were  more  familiar  with  the  standard  mouse  and  keyboard  (office 
setting;  mean  =  4.00)  than  with  any  of  the  other  devices.  With  one  exception,  all  comparisons 
with  the  standard  mouse  and  keyboard  (office  setting)  were  statistically  significant:  standard 
mouse  and  keyboard  (computer  gaming)  (mean  =  2.25,  t(ii)  =  4.98,  p  <  .01),  touchpad  with 
fingers  (mean  =  3.00,  tpi)  =  3.63,  p  <  .01),  and  touchpad  with  stylus  (mean  =  2.17,  tpp  =  6.16,  p 
<  .01).  The  difference  with  video  game  controllers  (mean  =  3.33,  tpp  =  1.87,  p  <  .10)  was  not 
significant. 


3.1  Task  Performance 

Tables  2  through  5  summarize  the  task  performance  comparisons  between  the  standard 
mouse  and  other  control  devices  for  each  task.  Task  performance  was  measured  by  average 
response  time  across  trials  for  the  neutral  point  movement,  dragging,  and  zooming  tasks.  Task 
perfonnance  for  the  tracking  task  was  measured  by  average  root  mean  square  error  across  trials. 

Sixteen  of  the  20  comparisons  between  the  standard  mouse  and  the  other  five  devices 
were  statistically  significant.  In  all  instances  where  the  difference  was  statistically  significant, 
the  standard  mouse  outperformed  the  other  devices  (i.e.,  lower  response  time,  smaller  RMS 
error).  The  standard  mouse  outperformed  all  other  devices  on  the  neutral  point  movement  and 
dragging  tasks.  It  outperfonned  the  Belkin  n52te,  Saitek  Cyborg  Command  Unit,  and  Xbox  360 
controller  on  the  tracking  task  and  the  Saitek  Cyborg  Command  Unit,  Wacom  Bamboo  Fun  with 
stylus,  and  Wacom  Bamboo  Fun  with  touch  on  the  zooming  task. 


Table  2.  Neutral  Point  Movement  Task  Average  Response  Time:  Means,  Standard 

Deviations,  and  t-tests. 


Device 

Mean 

SD 

Paired  Difl 

ferences  (Mouse  vs.  Other  Device) 

SE 

t 

df 

2-tailed 

Prob. 

Standard  Mouse 

1.17 

0.123 

Belkin  n52te 

3.71 

0.563 

0.148 

-17.167 

11 

.000 

Saitek  Cyborg 
Command  Unit 

2.88 

0.885 

0.227 

-7.521 

11 

.000 

Wacom  Bamboo 

Fun  with  Stylus 

1.76 

0.229 

0.056 

-10.487 

11 

.000 

Wacom  Bamboo 

Fun  with  Touch 

2.42 

0.293 

0.057 

-21.465 

11 

.000 

Xbox  360 

Controller 

2.68 

1.368 

0.373 

-4.038 

11 

.002 
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Table  3.  Tracking  Task  Average  Root  Mean  Square  Error:  Means,  Standard  Deviations, 

and  t-tests. 


Device 

Mean 

SD 

Paired  Difl 

ferences  (Mouse  vs.  Other  Device) 

SE 

t 

df 

2-tailed 

Prob. 

Standard  Mouse 

0.0612 

0.0173 

Belkin  n52te 

0.2948 

0.1236 

***** 

-6.649 

11 

.000 

Saitek  Cyborg 
Command  Unit 

0.4468 

0.1741 

***** 

-7.679 

11 

.000 

Wacom  Bamboo 

Fun  with  Stylus 

0.0924 

0.0839 

***** 

-1.344 

11 

.206 

Wacom  Bamboo 

Fun  with  Touch 

0.2274 

0.2867 

***** 

-2.029 

11 

.067 

Xbox  360 

Controller 

0.3564 

0.1976 

***** 

-5.221 

11 

.000 

Table  4.  Dragging  Task  Average  Response  Time:  Means,  Standard  Deviations,  and  t-tests. 


Device 

Mean 

SD 

Paired  Difl 

ferences  (Mouse  vs.  Other  Device) 

SE 

t 

df 

2-tailed 

Prob. 

Standard  Mouse 

1.64 

0.256 

Belkin  n52te 

5.04 

1.178 

0.312 

-10.905 

11 

.000 

Saitek  Cyborg 
Command  Unit 

4.54 

1.367 

0.356 

-8.135 

11 

.000 

Wacom  Bamboo 

Fun  with  Stylus 

2.18 

0.450 

0.186 

-2.904 

11 

.016 

Wacom  Bamboo 

Fun  with  Touch 

4.08 

0.992 

0.288 

-8.472 

11 

.000 

Xbox  360 

Controller 

4.75 

3.053 

0.857 

-3.626 

11 

.005 

11 
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Table  5.  Zooming  Task  Average  Response  Time:  Means,  Standard  Deviations,  and  t-tests. 


Device 

Mean 

SD 

Paired  Difi 

ferences  (Mouse  vs.  Other  Device) 

SE 

t 

df 

2-tailed 

Prob. 

Standard  Mouse 

4.30 

0.660 

Belkin  n52te 

4.42 

0.645 

0.166 

-0.713 

11 

.491 

Saitek  Cyborg 
Command  Unit 

4.870 

1.088 

0.194 

-2.928 

11 

.014 

Wacom  Bamboo 

Fun  with  Stylus 

5.15 

1.121 

0.222 

-3.840 

11 

.003 

Wacom  Bamboo 

Fun  with  Touch 

10.65 

1.584 

0.445 

-14.275 

11 

.000 

Xbox  360 

Controller 

4.02 

1.131 

0.255 

1.089 

11 

.299 

3.2  Subjective  Ratings 

Tables  6  through  9  summarize  the  subjective  ratings  comparisons  between  the  standard 
mouse  and  other  control  devices  for  each  task.  Ratings  were  on  5-point  scale:  1  -  very  poor,  2  - 
poor,  3  -  neutral,  4  -  good,  5  -  very  good.  Participants  rated  the  performance  of  the  standard 
mouse  equal  to  or  better  than  that  of  the  other  devices  on  all  4  tasks.  The  mouse  was  rated 
higher  than  all  other  devices  for  the  neutral  point  movement  task  and  higher  than  4  of  5 
alternative  devices  for  the  tracking  and  dragging  tasks  (the  exception  was  the  stylus  which  was 
not  significantly  different  from  the  mouse).  The  mouse  perfonnance  was  not  significantly 
different  from  the  other  devices  on  the  zooming  task. 
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Table  6.  Subjective  Ratings  of  the  Neutral  Point  Movement  Task:  Means,  Standard 

Deviations,  and  t-tests. 


Device 

Mean 

SD 

Paired  Difl 

ferences  (Mouse  vs.  Other  Device) 

SE 

t 

df 

2-tailed 

Prob. 

Standard  Mouse 

4.58 

0.669 

Belkin  n52te 

3.25 

1.289 

0.355 

3.752 

11 

.003 

Saitek  Cyborg 
Command  Unit 

3.67 

0.651 

0.260 

3.527 

11 

.005 

Wacom  Bamboo 

Fun  with  Stylus 

3.92 

0.669 

0.188 

3.546 

11 

.005 

Wacom  Bamboo 

Fun  with  Touch 

3.67 

0.778 

0.260 

3.527 

11 

.005 

Xbox  360 

Controller 

3.75 

0.965 

0.366 

2.278 

11 

.044 

Table  7.  Subjective  Ratings  of  1 

he  Tracking  Task:  Means,  Standard  Deviations,  and  t-te 

Device 

Mean 

SD 

Paired  Difl 

ferences  (Mouse  vs.  Other  Device) 

SE 

t 

df 

2-tailed 

Prob. 

Standard  Mouse 

4.33 

0.888 

Belkin  n52te 

1.50 

0.674 

0.297 

9.530 

11 

.000 

Saitek  Cyborg 
Command  Unit 

1.58 

0.996 

0.329 

8.370 

11 

.000 

Wacom  Bamboo 

Fun  with  Stylus 

3.83 

1.403 

0.435 

1.149 

11 

.275 

Wacom  Bamboo 

Fun  with  Touch 

3.00 

0.953 

0.376 

3.546 

11 

.005 

Xbox  360 

Controller 

2.17 

1.115 

0.458 

4.733 

11 

.001 
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Table  8. 


Device 

Mean 

SD 

Paired  Difl 

ferences  (Mouse  vs.  Other  Device) 

SE 

t 

df 

2-tailed 

Prob. 

Standard  Mouse 

4.58 

0.515 

Belkin  n52te 

3.17 

1.030 

0.288 

4.926 

11 

.000 

Saitek  Cyborg 
Command  Unit 

3.25 

1.055 

0.310 

4.304 

11 

.001 

Wacom  Bamboo 

Fun  with  Stylus 

4.08 

0.900 

0.230 

2.171 

11 

.053 

Wacom  Bamboo 

Fun  with  Touch 

3.42 

1.165 

0.241 

4.841 

11 

.001 

Xbox  360 

Controller 

3.33 

0.888 

0.305 

4.103 

11 

.002 

Table  9.  Subjective 

Ratings  of  the  Zooming  Task:  Means,  Standard  Deviations,  and  t-tei 

Device 

Mean 

SD 

Paired  Difl 

ferences  (Mouse  vs.  Other  Device) 

SE 

t 

df 

2-tailed 

Prob. 

Standard  Mouse 

3.83 

0.718 

Belkin  n52te 

3.67 

0.888 

0.271 

0.616 

11 

.551 

Saitek  Cyborg 
Command  Unit 

3.67 

0.985 

0.322 

0.518 

11 

.615 

Wacom  Bamboo 

Fun  with  Stylus 

3.17 

1.193 

0.449 

1.483 

11 

.166 

Wacom  Bamboo 

Fun  with  Touch 

3.50 

1.087 

0.414 

0.804 

11 

.438 

Xbox  360 

Controller 

4.08 

0.793 

0.279 

-0.897 

11 

.389 
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Figure  11.  Subjective  ratings  of  performance  for  each  task  by  device. 

Similar  results  were  observed  for  participants’  post-study  rankings  of  the  standard  mouse 
versus  the  alternative  devices  for  the  4  tasks.  The  standard  mouse  was  ranked  higher  than  the 
other  devices  on  12  of  20  comparisons.  The  mouse  performance  was  not  ranked  significantly 
different  from  the  other  devices  on  the  zooming  task.  A  summary  of  the  Wilcoxin  comparisons 
is  provided  in  Tables  B-l  through  B-4  and  Figures  B-l  through  B-5. 


4.0  DISCUSSION 

As  one  would  expect,  participants  performed  the  standard  move-and-click  and  dragging 
tasks  best  with  the  mouse.  Participants  indicated  a  high  level  of  experience  and  comfort  with  the 
mouse.  They  also  required  little  to  no  training  with  the  mouse,  as  they  had  perfonned  similar 
tasks  in  day-to-day  home  and  office  use.  The  mouse,  however,  did  not  provide  for  significantly 
better  performance  on  the  zooming  and  tracking  tasks  in  comparison  with  some  of  the  evaluated 
devices.  For  zooming,  the  Xbox  360  controller  and  Belkin  n52te  (which  featured  a  scroll  wheel 
similar  to  that  of  the  mouse)  were  not  significantly  different  from  the  standard  mouse  in 
perfonnance.  The  Wacom  Bamboo  Fun,  in  both  the  stylus  and  touch  conditions,  did  not  perfonn 
significantly  differently  than  the  mouse  on  the  tracking  task.  The  lack  of  significant  difference  in 
performance  between  the  mouse  and  Xbox  360  controller  on  the  zooming  task,  and  the  mouse 
and  Wacom  Bamboo  Fun  (utilizing  touch  or  stylus)  on  the  tracking  task,  is  of  interest  due  to  the 
lack  of  experience  and  minimal  training  with  each  device. 
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This  study  was  intended  to  relate  specific  attributes  of  each  device  to  the  individual  tasks 
at  which  they  excelled.  By  assessing  the  devices’  performance  levels  at  specific  tasks,  the 
origins  of  their  perfonnance  scores  can  be  assessed  and  the  features  most  successfully 
implemented  can  be  identified.  Participants  largely  identified  their  familiarity  with  the  mouse 
and  its  ergonomic  design  (comfort)  as  its  strengths.  For  simple,  repetitive  tasks,  such  as  the 
neutral  point  movement  and  dragging  tasks  (in  which  the  mouse  excelled),  it  is  important  to  use 
a  device  that  is  comfortable  and  feels  natural  in  order  to  avoid  fatigue.  Also  identified  as  a 
strength  of  the  mouse  was  the  ability  to  easily  control  the  cursor  at  whatever  speed  was  desired 
with  minimal  movement  and  concentration  required  by  the  user. 

Ergonomically,  the  Xbox  360  controller  was  rated  well  by  participants.  They  enjoyed  the 
ability  to  utilize  both  hands  and  relax  their  posture  with  the  controller  in  their  laps.  What  is  of 
interest  is  that  the  zooming  task  required  the  use  of  3  buttons  (zoom  in,  zoom  out,  accelerated 
zoom)  and  that  the  button  layout  and  design  on  this  device  provided  for  excellent  performance. 
The  “bumper”  buttons  (“RB”  and  “LB”)  seemed  advantageous  for  the  zooming  task  because 
they  allowed  the  participants  the  ability  to  comfortably  coordinate  use  of  their  index  fingers 
while  allowing  full  thumb  movement.  This  arrangement  allowed  for  excellent  bimanual 
coordination  to  manipulate  zoom  in  and  zoom  out  with  accelerated  zoom  for  quick  and  accurate 
inputs.  While  the  Belkin  n52te  also  perfonned  well,  it  utilized  the  same  feature  (a  scroll  wheel) 
for  the  zooming  task  as  was  present  on  the  standard  mouse.  Therefore,  the  lack  of  a  significant 
difference  in  the  performance  between  the  two  devices  on  the  zooming  task  is  expected. 

In  the  tracking  task,  the  Wacom  Bamboo  Fun  with  touch  operated  similarly  to  the  mouse 
using  relative  positioning — that  is,  one  can  pick  up  the  mouse  and  reposition  it  without  affecting 
the  cursor  on  the  screen.  By  contrast,  the  Wacom  Bamboo  Fun  with  stylus  operated  with 
absolute  positioning.  Absolute  positioning  is  implemented  on  the  Wacom  Bamboo  Fun  with  a 
rectangular  tablet/touchpad  that  serves  as  a  representation  of  the  screen.  As  such,  placing  the 
stylus  anywhere  on  the  touch  pad  would  snap  the  cursor  to  the  corresponding  position  on  the 
screen.  Comments  from  the  participants  indicated  that  using  either  touch  or  stylus  felt  natural 
and  familiar  for  use  in  the  tracking  task,  with  complaints  centering  on  a  limited  input  area  (the 
Wacom  Bamboo  Fun  tablet/touchpad).  It  is  likely  that  the  main  difference  between  the  three  top 
perfonners  (the  standard  mouse,  the  Wacom  Bamboo  Fun  with  touch,  and  the  Wacom  Bamboo 
Fun  with  stylus)  and  the  bottom  three  perfonners  (the  Xbox  360  controller,  the  Belkin  n52te,  and 
the  Saitek  Cyborg  Command  Unit)  in  the  tracking  task  is  related  to  the  amount  of  control  that 
participants  had  over  the  input  device.  The  lower  group  of  devices  utilized  analog  sticks  or  a  d- 
pad  for  movement  of  the  cursor.  The  upper  group  more  closely  emulated  direct  control  by 
allowing  participants  to  easily  manipulate  the  speed  of  the  cursor  movement  and  its  position. 

Participants’  comments  for  the  standard  mouse  noted  its  strengths  revolved  around  its 
familiarity  and  ease  of  use.  When  weaknesses  were  indicated,  they  focused  on  the  need  to 
reposition  the  mouse  when  it  reached  the  limits  of  the  mouse  pad  and  the  difficulty  in  performing 
the  zooming  task. 
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For  the  Belkin  n52te,  11  of  12  participants  commented  that  it  was  difficult  to  perform 
diagonal  movements  and/or  tracking.  Comments  for  the  Saitek  Cyborg  Command  Unit  focused 
on  ergonomic  issues  including  awkwardness  in  using  a  left-handed  device,  its  size  for  people 
with  small  hands,  and  difficulty  in  performing  tracking  with  the  thumb-driven  joystick.  For  the 
alternate  devices,  the  most  positive  comments  were  for  the  Wacom  Bamboo  Fun  either  with  the 
stylus  or  with  touch.  Participants  indicated  that  the  stylus  was  easy  to  use  and  effective  for 
tracking  and  dragging  tasks.  Negative  comments  regarding  the  stylus  indicated  that  it  was  too 
sensitive  and  was  not  suitable  for  the  zooming  task.  Participants  indicated  that  the  touch  mode 
was  easy  to  use,  but  as  with  the  stylus,  weaknesses  were  indentified  concerning  sensitivity  and 
calibration.  Finally,  participants  indicated  that  the  Xbox  360  controller  was  easy  to  use  and  that 
the  two-handed  configuration  was  desirable.  However,  there  were  some  concerns  with  using  the 
analog  stick  to  control  the  cursor.  Right-handed  participants  found  it  uncomfortable  to  use  their 
left  thumbs  to  operate  the  analog  stick  for  cursor  movement  and  thought  that  the  analog  stick  was 
too  sensitive  for  the  tracking  task. 

While  the  researchers  sought  to  limit  confounds  and  provide  an  even  field  for  testing  the 
devices  and  their  implemented  features  in  the  prescribed  tasks,  some  issues  still  arose  that  must 
be  mentioned.  Mapping  functions  to  the  various  devices  was  a  difficult  task  for  this  study.  It 
was  impossible  to  enable  all  devices  to  perform  to  their  fullest  capabilities  in  terms  of  movement. 
The  most  salient  example  of  this  was  the  Belkin  n52te.  Due  to  interaction  problems  between  the 
inner  workings  of  the  timing  algorithms  used  in  the  evaluation  software  and  the  internal  timers 
that  the  Belkin  n52te  uses  to  operate,  cursor  movement  with  the  device  was  a  difficult  task. 

While  vertical  and  horizontal  movement  worked  fairly  well,  diagonal  movement  was  severely 
hindered.  Any  movement  not  on  the  x-  or  y-axes  was  extremely  slow.  Participants  typically 
opted  for  a  path  involving  “up  and  over”  movement  rather  than  attempting  to  move  the  cursor 
diagonally,  as  they  would  have  done  with  a  more  capable  device. 

The  analog  sticks  for  the  Saitek  Cyborg  Command  Unit  and  the  Xbox  360  controller 
were  difficult  to  implement  effectively.  In  the  case  of  the  Saitek  device,  the  movement  of  the 
cursor  could  not  be  finely  calibrated.  Cursor  movement  was  slow  and  highly  sensitive,  rendering 
any  attempt  at  fine  movement  adjustments  an  arduous  task.  Our  implementation  of  the  Xbox 
360  controller  required  that  the  cursor  be  moved  at  a  fixed  speed  throughout  all  tasks.  In  most 
video  games  that  utilize  this  controller,  the  acceleration  is  set  so  that  fine  movements  are 
enabled.  This  meant  that  the  analog  stick  could  move  the  cursor  very  quickly,  but  that 
participants  could  not  make  fine  movements  easily.  The  Xbox  360  controller  also  had 
complications  with  cursor  registration.  Rather  than  registering  the  point  of  a  click  at  the  tip  of 
the  cursor  image  (an  arrow),  the  Xbox  360  controller  registered  it  at  the  bottom  of  the  image.  As 
a  consequence,  upon  clicking  the  circle  to  be  moved  in  the  dragging  task,  the  circle  would  often 
“drop”  at  the  bottom  of  the  cursor  image  instead  of  the  tip.  This  made  it  very  difficult  to  control 
where  the  dragged  circle  was  placed. 

It  would  seem  that  one  of  the  strengths  of  the  Wacom  Bamboo  Fun  with  touch  in  the 
zooming  task  would  be  the  ability  to  directly  influence  speed  of  the  zooming  movement  with  no 
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need  for  a  button  or  toggle  for  acceleration.  However,  the  Wacom  Bamboo  Fun  with  touch  did 
not  perfonn  as  expected  and  could  not  interpret  two-finger  touch  movements  at  the  speed  at 
which  they  were  perfonned.  Thus,  no  matter  how  quickly  one  attempted  to  zoom  in  or  zoom 
out,  the  on-screen  circle  would  enlarge  or  shrink  at  a  set  speed,  which  was  sluggish. 

Furthennore,  it  would  keep  any  movement  commands  in  a  queue.  If  participants  attempted  to 
have  it  zoom  very  quickly,  with  rapid  gesture  commands,  it  would  continue  the  zooming  motion 
even  after  they  ceased  providing  input.  This  lead  to  participants  “overshooting”  the  target  size 
and  was  a  large  source  of  frustration  for  participants. 

This  study  limited  the  implementation  of  the  gaming  keyboards.  The  Belkin  n52te  and 
Saitek  Cyborg  Command  Unit  were  intended  to  augment  the  performance  of  the  standard  mouse, 
rather  than  replace  it  and  control  the  cursor.  As  a  result,  these  devices  were  designed  to  be 
operated  using  the  left  hand  for  occasional  input  control.  Using  the  left-handed  gaming 
keyboards  as  the  primary  input  devices  for  the  tasks  studied  proved  difficult  for  a  majority  of  our 
participants,  as  10  of  12  indicated  that  they  were  right-handed.  This  means  that  device  efficiency 
was  not  maximized  because  participants  were  forced  to  use  their  non-dominant  hands. 

As  training  progressed  at  a  pace  controlled  by  the  participants,  there  was  no  metric  in 
place  to  determine  that  participants  were  comfortable  enough  with  a  device  to  perfonn  the 
study’s  tasks  adequately.  Thus,  participants  found  themselves  inadequately  prepared  in  some 
cases  due  to  rushing  through  the  training  tasks.  Participants  would  forget  which  inputs  were 
required  to  perform  the  task  at  hand  and  would  ask  the  test  administrator  for  confirmation  of  the 
controls  to  be  used.  In  future  research,  training  to  a  baseline  will  be  the  desired  method. 

Not  much  research  has  been  published  for  input  device  evaluation.  Most  research  is 
proprietary  to  manufacturers  and  not  available  in  the  public  domain.  For  that  reason,  this  study 
was  designed  incorporating  simple  devices  and  tasks.  Future  research  must  be  conducted  for 
more  complex  devices  in  a  task-saturated  environment.  One  area  of  research  that  is  of  interest 
would  be  evaluating  performance  in  bimanual  coordination  tasks  that  offer  more  complex  stimuli 
and  responses.  In  this  way,  one  could  more  closely  assess  the  full  range  of  human  manual  input 
potential.  This  could  involve  the  use  of  touch  displays  that  offer  greater  freedom  of  movement 
and  direct  input,  rather  than  exclusively  evaluating  indirect  input  devices  as  was  done  in  this 
study. 

The  gaming  keyboards  likely  have  uninvestigated  potential  when  used  in  concert  with  the 
mouse,  as  designed.  Due  to  the  limited  evaluation  of  the  gaming  keyboard  devices  in  this  study, 
further  research  into  their  use  would  be  required  to  adequately  assess  their  full  capabilities.  The 
configuration  of  a  left-handed  gaming  keyboard  paired  with  a  mouse  could  also  be  of  interest  in 
a  bimanual  coordination  study. 

New  technologies  arise  every  day.  These  devices  provide  unforeseen  capabilities  and 
potential  applications.  For  this  reason,  input  device  evaluations  must  take  an  iterative  approach 
which  constantly  implements  and  assesses  new  devices  and  compares  them  to  existing 
technologies.  Input  devices  are  currently  introduced  to  the  market  with  a  variety  of  feature 
combinations.  The  consumer  must  make  assumptions  as  to  what  performance  these  devices 
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provide  in  different  contexts.  Input  device  assessments  allow  the  customers  to  not  only  better 
select  the  device  required  for  their  specific  needs,  but  also  enable  them  to  have  more  robust 
requirements  so  that  devices  can  be  created  that  would  explicitly  support  their  unique  task 
environments.  In  this  case,  devices  that  aid  an  operator  in  supervisory  control  of  multi-RPAs 
may  be  developed  that  provide  new  capabilities  and  enhance  perfonnance. 
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Appendix  A 
Study  Questionnaires 
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Demographic  Questionnaire 


Is  your  vision  20/20  or  correctable  to  20/20? 

Yes 

No 

What  is  your  handedness? 

Right 

Left 

Ambidextrous 

In  what  age  group  are  you? 

18-25 

26-35 

36-45 

Please  rate  your  expertise  with  video  games  (circle  one): 

None  I  never,  or  rarely,  play  video  games 

Novice  I  occasionally  play  video  games 

Average  I  have  my  own  video  game  system  that  I  use  once  or  twice  a  week 

Experienced  I  enjoy  playing  video  games  often  and  can  learn  new  games  easily 

Expert  I  am  very  skilled  with  my  favorite  types  of  games  and  play  them  a  lot 


Rate  computer/video  game  experience  with  the  following  systems  ( I  =  Very  Low,  2=Low, 
3=Average,  4=High,  5=Very  High): 

Standard  mouse  and  keyboard  (office  setting)  1 

Standard  mouse  and  keyboard  (computer  gaming)  1 


Touch  screens  (used  with  fingers) 
Touch  screens  (used  with  stylus) 
Video  game  controllers 


2 

2 

2 

2 

2 


3 

3 

3 

3 

3 


4 

4 

4 

4 

4 


5 

5 

5 

5 

5 
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Post-Trial  Questionnaire 


For  questions  1-4,  please  rate  the  device ’s  performance  on  a  scale  of  1-5  for  each  of  the  given 
tasks  (1  being  ‘very  poor’,  3  being  ‘neutral’,  5  being  ‘very  good’)  and  add  any  comments  you 
may  have. 

Circle  the  device  you  are  evaluating: 

1 )  Mouse 

2)  Belkin  n52te 

3)  Sartek  Cyborg  Command  Unit 

4)  Wacom  Bamboo  Fun  with  Stylus 

5)  Wacom  Bamboo  Fun  with  Touch 

6)  Xbox  360  Controller 

1 .  How  well  do  you  feel  this  device  helped  you  perform  the  neutral  point  movement  task? 

1  2  3  4  5 

2.  How  well  do  you  feel  this  device  enabled  you  to  perform  the  tracking  task? 

1  2  3  4  5 

3.  How  well  do  you  feel  this  device  enabled  you  to  perform  the  dragging  task? 

1  2  3  4  5 

4.  How  well  do  you  feel  this  device  enabled  you  to  perform  the  zooming  task? 

1  2  3  4  5 

5.  What  were  the  weaknesses  of  this  device  for  performance  in  all  of  the  tasks? 
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6.  What  were  the  strengths  of  this  device  for  performance  in  all  of  the  tasks? 


7.  What  modifications  would  you  make  to  this  device  (configuration,  design,  button 
functions,  etc.)?  Provide  illustrations  if  necessary. 
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Post-Session  Questionnaire 

1 .  Rank  the  following  controllers  in  order  of  perfonnance  for  the  neutral  point  movement 
task  (1-6,  1  being  best  performance): 


Mouse 

Belkin  n52te 

Saitek  Cyborg  Command  Unit 

Bamboo  Fun  tablet  (stylus) 

Bamboo  Fun  tablet  (touch) 

Xbox  360  Controller 

2.  Rank  the  following  controllers  in  order  of  perfonnance  for  the  dragging  task  (1-6,  1  being 
best  perfonnance): 


Mouse 

Belkin  n52te 

Saitek  Cyborg  Command  Unit 

Bamboo  Fun  tablet  (stylus) 

Bamboo  Fun  tablet  (touch) 

Xbox  360  Controller 

3.  Rank  the  following  controllers  in  order  of  perfonnance  for  the  tracking  task  (1-6,  1  being 
best  perfonnance): 
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Mouse 

Belkin  n52te 

Saitek  Cyborg  Command  Unit 

Bamboo  Fun  tablet  (stylus) 

Bamboo  Fun  tablet  (touch) 

Xbox  360  Controller 

4.  Rank  the  following  controllers  in  order  of  perfonnance  for  the  zooming  task  (1-6,  1  being 
best  perfonnance): 


Mouse 

Belkin  n52te 

Saitek  Cyborg  Command  Unit 

Bamboo  Fun  tablet  (stylus) 

Bamboo  Fun  tablet  (touch) 

Xbox  360  Controller 

5.  What  were  your  favorite  aspects  of  any  of  the  controllers? 


6.  What  were  your  least  favorite  aspects  of  any  of  the  controllers? 
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7.  What  would  you  change,  combine,  or  take  away  from  any  of  the  controllers  to  aid 
perfonnance  in  the  prescribed  tasks? 


8.  Are  there  any  other  considerations  you  wish  to  bring  up  to  the  test  administrator? 
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Appendix  B 

Summary  of  Wilcoxin  Signed  Ranks  Tests 
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Table  B-l.  Subjective  Rankings  on  the  Neutral  Point  Movement  Task:  Standard  Mouse  vs. 
Alternative  Devices  -  Wilcoxin  Signed  Ranks  Test 


Comparison 

N  Ranks 

Z 

2-Tailed 

Probability 

Standard  Mouse  vs.  Belkin 

Positive  Ranks 

11 

-2.957 

.003 

n52te 

Negative  Ranks 

1 

Standard  Mouse  vs.  Saitek 

Positive  Ranks 

11 

-2.721 

.007 

Cyborg  Command  Unit 

Negative  Ranks 

1 

Standard  Mouse  vs.  Wacom 

Positive  ranks 

8 

-0.996 

.319 

Bamboo  Fun  with  Stylus 

Negative  Ranks 

4 

Standard  Mouse  vs.  Wacom 

Positive  Ranks 

10 

-1.764 

.078 

Bamboo  Fun  with  Touch 

Negative  Ranks 

2 

Standard  Mouse  vs.  Xbox  360 

Positive  Ranks 

9 

-2.082 

.039 

Controller 

Negative  Ranks 

3 

Table  B-2.  Subjective  Rankings  on  the  Tracking  Task:  Standard  Mouse  vs.  Alternative 
Devices  -  Wilcoxin  Signed  Ranks  Test 


Comparison 

N  Ranks 

Z 

2-Tailed 

Probability 

Standard  Mouse  vs.  Belkin 

Positive  Ranks 

12 

-3.105 

.002 

n52te 

Negative  Ranks 

0 

Standard  Mouse  vs.  Saitek 

Positive  Ranks 

12 

-3.078 

.002 

Cyborg  Command  Unit 

Negative  Ranks 

0 

Standard  Mouse  vs.  Wacom 

Positive  ranks 

7 

-0.855 

.392 

Bamboo  Fun  with  Stylus 

Negative  Ranks 

5 

Standard  Mouse  vs.  Wacom 

Positive  Ranks 

10 

-2.496 

.013 

Bamboo  Fun  with  Touch 

Negative  Ranks 

2 

Standard  Mouse  vs.  Xbox  360 

Positive  Ranks 

12 

-3.104 

.002 

Controller 

Negative  Ranks 

0 
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Table  B-3.  Subjective  Rankings  on  the  Dragging  Task:  Standard  Mouse  vs.  Alternative 
Devices  -  Wilcoxin  Signed  Ranks  Test 


Comparison 

N  Ranks 

Z 

2-Tailed 

Probability 

Standard  Mouse  vs.  Belkin 

Positive  Ranks 

12 

-3.090 

.002 

n52te 

Negative  Ranks 

0 

Standard  Mouse  vs.  Saitek 

Positive  Ranks 

11 

-2.999 

.003 

Cyborg  Command  Unit 

Negative  Ranks 

1 

Standard  Mouse  vs.  Wacom 

Positive  ranks 

9 

-2.174 

.030 

Bamboo  Fun  with  Stylus 

Negative  Ranks 

3 

Standard  Mouse  vs.  Wacom 

Positive  Ranks 

11 

-2.338 

.019 

Bamboo  Fun  with  Touch 

Negative  Ranks 

1 

Standard  Mouse  vs.  Xbox  360 

Positive  Ranks 

10 

-2.285 

.022 

Controller 

Negative  Ranks 

2 

Table  B-4.  Subjective  Rankings  on  the  Zooming  Task:  Standard  Mouse  vs.  Alternative 
Devices  -  Wilcoxin  Signed  Ranks  Test 


Comparison 

N  Ranks 

Z 

2-Tailed 

Probability 

Standard  Mouse  vs.  Belkin 

Positive  Ranks 

9 

-1.303 

.192 

n52te 

Negative  Ranks 

3 

Standard  Mouse  vs.  Saitek 

Positive  Ranks 

9 

-1.711 

.087 

Cyborg  Command  Unit 

Negative  Ranks 

3 

Standard  Mouse  vs.  Wacom 

Positive  ranks 

6 

-0.831 

.406 

Bamboo  Fun  with  Stylus 

Negative  Ranks 

6 

Standard  Mouse  vs.  Wacom 

Positive  Ranks 

4 

-0.079 

.937 

Bamboo  Fun  with  Touch 

Negative  Ranks 

8 

Standard  Mouse  vs.  Xbox  360 

Positive  Ranks 

6 

-0.158 

.874 

Controller 

Negative  Ranks 

6 
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Neutral  Point  Tracking  Dragging  Zooming 

Movement 


■  Mouse 

■  Belkin 


Figure  B-l.  Participants’  preferences  for  standard  mouse  versus  Belkin  n52te  by  task. 


Neutral  Point  Tracking  Dragging  Zooming 

Movement 


■  Mouse 

■  Saitek 


Figure  B-2.  Participants’  preferences  for  standard  mouse  versus  Saitek  Cyborg  Command 
Unit  by  task. 
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Neutral  Point  Tracking  Dragging  Zooming 

Movement 


■  Mouse 

■  Stylus 


Figure  B-3.  Participants’  preferences  for  standard  mouse  versus  Wacom  Bamboo  Fun  with 
stylus  by  task. 


Neutral  Point  Tracking  Dragging  Zooming 

Movement 


■  Mouse 

■  Touch 


Figure  B-4.  Participants’  preferences  for  standard  mouse  versus  Wacom  Bamboo  Fun  with 
touch  by  task. 
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Neutral  Point  Tracking  Dragging  Zooming 

Movement 


p  Mouse 
■  Xbox  360 


Figure  B-5.  Participants’  preferences  for  standard  mouse  versus  Xbox  360  controller  by 
task. 
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